Deriving Synthetic Validity Models: Is R = .80 Large Enough?

Many view the .50-.80 Rs typically seen for job-component validation (JCV) models as adequate to justify inferences of validity/utility. This study derived JCV models predicting DOT Strength from O*NET dimensions; although a R = .82 was achieved, errors of prediction were too large to justify its applied use. O*NET has received considerable recent attention in the context of developing methods of " synthetic " test validation such as the job-component validation (JCV) technique (e. have advanced optimistic conclusions regarding the quality and utility of O*NET data, including its suitability as a data source for quantifying the job activities and demands that are used as predictors when deriving JCV models that predict worker attribute requirements. For example, LaPolice et al. (2008) showed that when predicting the criterion of occupational literacy levels, JCV models based on O*NET data produced multiple Rs ranging from .80 to .83. Although other JCV studies (e.g., Jeanneret & Strong, 2003) have reported derivation-sample Rs that are much lower in magnitude than those found by LaPolice et al. (ranging as low as the .30's, .40's and .50's), JCV proponents remain highly upbeat in their assessments of the potential of JCV to revolutionize the test validation process. For example, Johnson, Steel, Scherbaum, Hoffman, Jeanneret, and Foster (2010) asserted that although " there are still several details to hammer out, especially the number of job components to be used [as predictors] and possibly modifying the O*NET to include aspects gleaned from PAQ-based synthetic validity successes .... there are no theoretical obstacles left " (p. 325, emphasis added), and that JCV " has great potential to advance the science and practice of industrial and organizational psychology " (p. 305, emphasis added). LaPolice et al. (2008) concluded that " these multiple Rs are all very close to the maximum correlation for each dependent variable, suggesting that our models are approaching the best possible prediction " (p. 435, emphasis added). They went on to conclude that " the fact that the multiple Rs are all close to the maximum possible provides evidence for the construct validity of O*NET " (p. 435, emphasis added), and that " in summary, this study suggests that O*NET descriptor data predict literacy scores with a high degree of accuracy, providing evidence for the construct validity of O*NET " (p. 437, emphasis added). Jeanneret and Strong (2003) were similarly optimistic regarding the suitability of O*NET and JCV for mission-critical …